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The 120° ordered antiferromagnetic state of the Hubbard model on a triangular lattice presents an 
interesting case of [/-controlled competing interactions and frustration. The spin stiffness is found 
to vanish at U* tiS « 6 and the spin- wave energy ujm at q M = (2n/3, 0) etc. is found to vanish at 
[TJf ~ 6.8 due to competing spin couplings generated at finite U. The loss of magnetic order due to 
the magnetic instability at qM yields a first-order quantum phase transition in the insulating state 
at U = E/jJi- Implications of the quantum spin disordered insulator to the spin-liquid state and Mott 
transition in the organic systems k — (BEDT — TTF)2X are discussed. Effects of hole and electron 
doping on magnetic ordering and spin stiffness are also examined. 



I. INTRODUCTION 

Recent 1 H NMR and static susceptibility measure- 
ments on the nearly isotropic, triangular-lattice antifer- 
romagnet k - (BEDT - TTF) 2 Cu 2 (CN) 3 have shown no 
indication of long-range magnetic ordering down to 32 
mK, well below the estimated exchange constant J ~ 250 
K, suggesting the realization of a quantum spin-liquid 
statei No signature of antiferromagnetic (AF) transition 
was seen in earlier EPR measurements as well^ 

On the other hand, a non-colinear 120° ordered ground 
state is the accepted consensus for the S =1/2 quantum 
Heisenberg antiferromagnet (QHAF) on an isotropic tri- 
angular latticed Recent quantum Monte Carlo calcu- 
lations yield a quantum reduction of 59% to the mag- 
netic order from its classical valued quite close to the 
spin- wave theory result of 52% in the first-order 1/5 
expansion?^ which is somewhat greater than the 40% 
reduction for the unfrustrated square-lattice. 

In this paper we show that the absence of long-range 
magnetic order in the nearly isotropic organic antifer- 
romagnet k - (BEDT - TTF) 2 Cu 2 (CN) 3 can be un- 
derstood in terms of strongly enhanced quantum spin 
fluctuations due to finite [/-induced competing interac- 
tions and frustration. Indeed, for the half-filled Hubbard 
model on an isotropic triangular lattice, we find that in 
the insulating ordered state the spin stiffness vanishes 
at U* tiS ~ 6. The corresponding divergence in the spin- 
fluctuation correction implies loss of long-range magnetic 
order at U* rdeT which is somewhat higher than i7* tiff , 
yielding a fmitc-CZ magnetic quantum phase transition 
in the insulating state. Even deeper in the insulating 
state, we find an instability of the 120° ordered state at 
U^j = 6.8 against out-of-plane spin fluctuations of wave 
vector q^f = (2ir/3, 0) etc., implying instability towards 
a F-AF state. Since spin fluctuations remain finite as 
U — ► U^j, the magnetic instability implies a first-order 
quantum phase transition, thus pre-empting the vanish- 
ing spin-stiffness instability. 

The realization in the triangular lattice of a non- 
magnetic insulator state at intermediate U, in which 
magnetic ordering is suppressed by strong quantum spin 
fluctuations, is interesting as it allows, with decreasing 



U , for a Mott- type metal- insulator transition not accom- 
panied by any magnetic symmetry breaking. In this con- 
text, another quantum correction which assumes signif- 
icance is that for the AF band gap. Indeed, two differ- 
ent scenarios emerge depending on the relative magni- 
tudes of U* A „ and U* , where the 120° AF order and 

oraci S^P ' 

AF band gap vanish, respectively. For U* TdeI > U* &p , a 
non-magnetic insulator state lies between the paramag- 
netic metal (PM) and the AF insulator (AFI), whereas 
for ?7* ap > U* rdcl , the magnetic transition is pre-empted, 
and there is a (nearly) first-order transition from AFI to 
PM at U* when the two AF bands start overlapping, 
possibly with an intervening antiferromagnetic metallic 
(AFM) phase in a narrow U range. Quantum corrections 
to quasiparticle dispersion and band gap due to motion 
of an added hole (electron) in the AF background, re- 
sulting in strong incoherence due to scrambling of the 
spin ordering, has been recently studied in detail for the 
t — i'-Hubbard model on a square latticed 

The divergent quantum spin fluctuations due to van- 
ishing spin stiffness and the magnetic instability towards 
a F-AF state may actually be a precursor to an ex- 
otic quantum spin-disordered state, as for the frustrated 
square-lattice antiferromagnet. Indeed, for the spin- 
half J — J' Heisenberg model on a square lattice, or 
equivalently the strong-coupling t — i'-Hubbard model, 
where the frustrating NNN coupling J' — At' 2 /U leads 
to vanishing spin stiffness and a (0, n) instability to- 
wards a F-AF state at J'/ J = t' 2 /t 2 = 0.5,2 series- 
expansion studie a 1 ^ 11 of the ground-state energy indi- 
cate a continuous transition at J'/ J ~ 0.4 from the AF 
state which breaks spin-rotation symmetry to a colum- 
nar dimer (valence-bond-solid) state which breaks lattice 
translation symmetry, although it is controversial^ The 
location of the transition at J'/ J ~ 0.4 is surprisingly 
very close to where the AF order also vanishes^ Recently 
there has been strong interest in the critical theory of con- 
tinuous quantum phase transitions between two phases 
with different broken symmetry, which requires going be- 
yond the Landau-Ginzburg- Wilson paradigm^ 

The suppression of magnetic ordering due to enhanced 
spin fluctuations is therefore also relevant in the lay- 
ered system k - (BEDT - TTF) 2 Cu[N(CN) 2 ]Cl, which 
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exhibits a genuine Mott transition not accompanied by 
any symmetry breakingjM Generally, the organic systems 
k— (BEDT — TTF)2X, where X denotes inorganic mono- 
valent anion, have emerged as a new class of correlated 
electron systems exhibiting antiferromagnetism, super- 
conductivity, and metal-insulator transitioni^i& Recent 
discovery of superconductivity in Na x Co02-yH20i£ and 
the observation of low-temperataure insulating phases 
in some VS-adlayer structures such as K on Si[lll]r& 
have also renewed interest in correlated electron system 
on triangular lattices. As a recent example of quasi- 
two-dimensional antiferromagnetism on a triangular lat- 
tice exhibiting the 120° spin ordering, RbFe(Mo04)2 has 
been studied using elastic neutron scattering^ and mag- 
netic resonance and magnetization experiments^ 

The Hubbard model on a triangular lattice has been 
studied recently using a variety of tools. The non- 
magnetic insulating state near the Mott transition has 
been studied using the path integral renormalization 
group method, 21 in which the HF results are systemat- 
ically improved to reach the true ground state by tak- 
ing account of quantum fluctuations. Results show a 
generic emergence of a non-magnetic insulating state 
sandwiched by a Mott metal-insulator transition and an 
AF transition. The zero-temperature phase diagram has 
been studied using the slave boson technique and the ex- 
act diagonalizationiSS The mean-field SB approach yields 
a rich phase diagram qualitatively resembling the HF 
resultsiS&Si One-electron density of states has been ex- 
amined using the quantum Monte Carlo method* 2 ^ show- 
ing a pseudogap development for intermediate U, accom- 
panied by two peaks in the spin structure factor, signal- 
ing the formation of a spiral spin density wave (SDW). A 
weak-coupling RG analysis applied to the anisotropic tri- 
angular lattice shows that frustration suppresses the an- 
tiferromagnetic instability in favour of a superconducting 
instability 26 A magnetic field induced exotic spin-triplet 
superconductivity has been proposed having strong fer- 
romagnetic fluctuations £L Ground-state spin structure of 
Cr and Mn monolayers on Cu[lll], proposed as ideal can- 
didates for physical realization of frustrated 2D itinerant 
antiferromagnets, has been investigated by performing ab 
initio calculations based on the density-functional theory 
in the local spin-density approximation^ 

A spin-liquid type non-magnetic insulating (NMI) 
state sandwiched between a weak-coupling PM state 
and a strong-coupling AFI state has also been obtained 
for the t — t'-Hubbard model on a square lattice and 
an anisotropic triangular lattice using the path integral 
renormalization group methodi22^a The NMI state has 
been recently suggested to be a new type of degener- 
ate quantum spin phase having gapless and dispersion- 
less spin excitations^ At the same time, this result of 
an intervening NMI state is in contradiction to the ear- 
lier finding of an intermediate metallic antifcrromagnetic 
state ( AFM) i^I We briefly compare the two types of lat- 
tices to highlight the common and distinguishing features 
of the AF state. 



Frustration suppresses the spin stiffness in both cases, 
rendering the AF state more susceptible to loss of mag- 
netic order due to quantum fluctuations. While frustra- 
tion is explicitly controlled by t' in the square lattice 
and can be tuned to zero, the triangular lattice pro- 
vides a subtle case of [/-controlled intrinsic frustration 
where the stiffness can be tuned to zero at finite U. An- 
other common consequence of frustration is the AF band 
broadening, which is due to same-sublattice hopping t' 
in the square lattice and due to the spiral ordering in the 
triangular lattice. The consequent reduction of the AF 
band gap also enhances the fluctuation correction due 
to interband transfer of spectral weight. Furthermore, 
the suppression of the perfect-nesting instability in both 
cases has the result that magnetic ordering sets in only 
above a critical interaction strength. Whether the onset 
of magnetic ordering and opening of AF band gap occur 
at different U values, yielding an intervening AFM phase 
for the triangualr lattice, is a pertinent question. At the 
HF level, an AFM phase is indeed stabilized, although 
in a narrow U range 4.7 < U < 5.1, indicating that 
the magnetic transition is nearly first order; whereas for 
the square lattice with if ft < 0.5, the magnetic order col- 
lapses when the two AF bands start overlapping, yielding 
a strictly first-order magnetic transition, the AFM phase 
is again stabilized for the cubic lattice M£l£l This shows 
a subtle dependence on the density of overlapping band 
states, indicating that density of states renormalization 
due to quantum fluctuations should have an important 
effect on stabilization of the AFM phase. 

We consider the Hubbard model 

H = -t^2al„a i+5ia + a l] a ^ a li a H (!) 

i,S i 

with nearest-neighbour (NN) hopping on a triangular lat- 
tice, and focus on the effect of finite [/-induced competing 
interactions on quantum spin fluctuations in the 120° or- 
dered state. The mean-field state is briefly reviewed in 
section II to introduce the notation. We obtain the spin 
fluctuation propagator (section III) and study the spin- 
wave excitations and spin stiffness in the full U range 
(section IV). We also examine the effects of hole and 
electron doping on magnetic ordering (section V). 



II. MEAN-FIELD STATE 

There are two alternative mean-field descriptions of the 
120° ordered AF state — i) a spiral-state representation, 
with an ordering wavevector Q = (2n/3, 2n/\/3), and 
ii) a three-sublattice representation, involving the local 
mean fields A Q on the three sublattices a = A, B, C. The 
energy eigenvalues and eigenvectors of the sublattice- 
basis [6 x 6] Hamiltonian matrix can be conveniently ob- 
tained from those of the spiral-state [2 x 2] Hamiltonian, 
as described below. 
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A. Spiral-state representation 

With an order parameter Aq = — U ^2 k {a k _Q\ dki), 
representing spin ordering in the x~y plane, the Hubbard 
Hamiltonian reduces to 



H, 



HF 



= £(4 



kT fl k-Qi 



£k Aq 

A Q e k _ Q 



ak| 

Ok-Qi 



(2) 



at the Hartree-Fock (HF) level, where ek = —2t[cosk x + 
2cos(k x /2) cos(k y V3/2)] is the triangular-lattice free- 
fermion energy. Choosing real Aq , the spiral-state quasi- 
particle energies E^ and amplitudes («k Vk) are given by 



(3) 




(4) 



for the upper (+) and lower (— ) AF bands, where r?k = 
(ek + £k-Q)/2 and £ k = (^k - £k-cj)/2. Self-consistency 
requires that Aq = — C/^k w k w k, which yields the con- 
dition 



u 4^ 



l 



:[9{E F - E-) - e(E ¥ - E+)] (5) 



in which lower and upper band states contribute with 
opposite sign. 

In terms of the fcrmion spinor \&j = (a^ ctjj), the spin 
expectation values (Sf ) = ^(^ja^^i) in the spiral state 
yield 



( s i) = ^^QCOsQ.ri 
(Si) = ^TOQsinQ.ri 



(6) 



at lattice site i, where the spiral-state magnetization 
m Q = 2Ek( a k-Q|«kT>- For Q = (2tt/3, 2tt/V3) the 
spiral twisting of spins generates the 120° ordered AF 
state on the triangular lattice. 



B. Three-sublattice representation 

While the spiral-state description applies only to Bra- 
vais lattices, the sublattice-basis description applies to 
Kagome type non-Bravais lattices as well. In the Hartree- 
Fock approximation, the Hamiltonian reduces to 

WHF = 2*![-°'- A <]*<-*E*. tl *<+*' ( 7 ) 

i i.S 



where the local mean field Aj = {^\a^ i) . In general, 
the 120° AF state is characterized by an ordering plane 
(normal hi) and a planar direction {112) with reference 
to which Aj make angles 6 a = 0°, 120°, and 240° on 
the three sublattices a — A, B, C. A convenient choice is 
hi = z (spin-ordering in the x — y plane) and hi = x, so 
that Aj on the three sublattices are given by 



A„ = Ad (a = a, b, c) 
in terms of the three lattice unit vectors 



(8) 



1 



V3\ 
-Try-, 



1 



V- (9) 



In general, in terms of the two mutually perpendicular 
directions hi and hi, the spin orientations in Eq. (8) are 
given by 



a = cos9 a hi + sin8 a (hi x hi). 



(10) 



As Aj is identical on sites of the same sublattice, 
Fourier transformation within the sublattice basis yields 
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(11) 



Here * k = (ak| «kj. &k| &ki c k | c k j.), where a k , K, Ck are 
fermion operators defined on the three sublattices A, B, 
C. Wavevector k lies within the Magnetic Brillouin Zone 
(MBZ), corresponding to the three inter-penetrating tri- 
angular sublattices (lattice parameter \/3a). The NN 
hopping term 



5— d,6,c 



ik.S 



-t[e lk * + 2e- lk */ 2 cos(V3fc y /2)] 

(12) 

mixes AB, BC, and CA sublattices, which are connected 
by the three lattice unit vectors. 

The six eigenvalues E kl and eigenvectors Ik*, I) of the 
[6 x 6] Hamiltonian matrix in Eq. (11), corresponding to 
upper (+) and lower (— ) AF bands, follow from Eqs. (3) 
and (4) for the spiral-state Hamiltonian. Here I = 1,2,3 
refer to the three branches corresponding to momentum 
values k, k + Q, k — Q, respectively, k being restricted 
to the MBZ. The amplitude |k, l) a involves not only the 
spin orientation <p a corresponding to sublattice a, but 
also a relative phase angle <5q" associated with the spiral 
twisting, and is given by 



|k, I) a — 



e Q 



(13) 



where the planar spin orientations <p a =0°, 120°, 240° 
for the three sublattices. The spiral phase angle 



5 1 q = 0, ±Q.R a A 



(14) 
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(relative to A) for I = 1,2, 3, where R q a is the primitive 
lattice vector connecting sublattices a and A. In view of 
above structure of state |k), the self-consistency condi- 
tion TO^ = 2A£/U = J2u il^lk, l) a retains the same 
form as in Eq. (5). 



III. SPIN FLUCTUATIONS 

Associated with the spontaneous symmetry breaking 
of the continuous spin-rotation symmetry of the Hubbard 
model, there exist gapless transverse fluctuation modes or 
Goldstone modes, involving fluctuations locally normal 
to the symmetry-breaking directions. These low-energy 
collective excitations, studied earlier in the context of 
doped cuprates^ play an important role in determin- 
ing several important physical properties in ordered sys- 
tems, such as stability of the mean-field ordered state, 
quantum corrections to the order parameter and ground- 
state energy, temperature dependence of the order pa- 
rameter in three dimensions, and renormalization of elec- 
tron (hole) spectral function and AF band gap due to 
fermion-magnon scattering of mobile carriers in the mag- 
netic background. 

We consider the time-ordered spin fluctuation prop- 
agator, the Fourier transform of which is given in the 
sublattice basis as 



iq,(ri-r 3 ) 



3 

x (* G |T[s?(t)s;(0]* G ) 



(15) 



where a, (3 = A, B, C are the sublattice indices and \i, v = 
x,y,z are the spin directions. To leading order in an 
inverse-degeneracy expansion, 34 equivalent to summing 
over all bubble diagrams, 



[x(q,^)] 



l[x°(q^)] 
l-C7[ X °(q,w) 



(16) 



where the bare particle- hole propagator [x°(q, w)] is ob- 
tained by integrating out the fermions in the broken- 
symmetry state. For the half-filled insulating state, the 
added hole (particle) states lie in the lower (upper) Hub- 
bard band, and we have 



[x°(q-)C^E^ 



— * 



2 ^ E? 

Mm k-q 



(17) 

where (c M ) Q + denotes the particle-hole spin matrix ele- 
ment on the a sublattice 
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FIG. 1: The spin- wave dispersion along different symmetry 
directions of the MBZ (inset), for A = 4 (U = 8.8). 

A q (cc>) and eigenvectors |0q(w)) of which contain informa- 
tion regarding the massless transverse fluctuations (spin 
waves) as well as the massive longitudinal fluctuations 
along the local ordering direction. Also included are the 
particle-hole (Stoner) excitations across the band gap, 
given by the poles of [x°(q, w)]^. In the following, we 
focus on the spin- wave energies w q , which are obtained 
from the poles 1 — J7A q (w q ) = of Eq. (16). 

Evaluating and diagonalizing the \x (q, u>)] matrix for 
q, u> = 0, we obtain three Goldstone modes UX n = 1, as 
expected for the non-colinear 120° AF state, correspond- 
ing to rigid spin rotations about the x, y, z axes. Rotation 
around the z axis yields an in-plane mode involving S x 
and S y fluctuations, while those around the x and y axes 
yield two out-of-plane modes involving only S z fluctua- 
tion. The corresponding eigenvectors, giving transverse 
fluctuation amplitudes on the three sublattices, are 




and 



(19) 



For small q, oj, the Goldstone mode eigenvalue A q (w) 
has the following typical form for the AF state: 



Aq 2 + Bcu 2 



(20) 



where the coefficient A of the q 2 term is proportional 
to the spin stiffness p. The pole equation 1 — UX = 
yields the spin-wave energy w q = cq, where the spin- 
wave velocity c = \J A/B oc J~p is related to the spin 
stiffness. 



IV. SPIN- WAVE SPECTRUM 



{<J^-+ = <k-q+,mK|k-,0 Q . 
The bare particle- hole propagator [x° (q, oj) 



la,f3 



(18) 
is a [9 x 



9] Hermitian matrix in the three sublattice (a=A,B,C) 
and three spin (/i = x, y, z) basis, the (real) eigenvalues 



The full spin- wave dispersion along different symmetry 
directions in the Magnetic Brillouin Zone is shown in Fig. 
1 for U = 8.8, corresponding to a self-consistent mean- 
field A = 4. Along the T-K direction, the softer mode 
is doubly degenerate, and the dispersion continues along 
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FIG. 2: The spin-wave energy luk at the Magnetic Brillouin 
Zone vertices K, along with the strong-coupling result 3JS. 



FIG. 3: The spin-wave energy ujm at the mid-points (M) of 
the MBZ edges, along with the strong-coupling result 2JS. 



the K-M direction after crossing at the MBZ vertex K, 
where all three modes become degenerate. The degenera- 
cies along T-K and K-M directions in k space are simply 
related to the two equivalent 120° orderings (clockwise 
and anticlockwise rotation of spins) along the correspond- 



ing orthogonal lattice directions x and 



The 



degeneracy is resolved along the M-r direction, but two 
modes again become degenerate near T, and these repre- 
sent the two out-of-plane (S z ) fluctuation modes. Near 
the r point (q << 1), the z sector does not mix with the 
x — y sector in [x°(q, w)], and the sublattice symmetry in 
z sector yields the two degenerate S z fluctuation modes 
given in Eq. (19). Behaviour of w q near the special MBZ 
points r, K, and M is further discussed below. 

We next consider the spin-wave energy at the MBZ 
vertices K with q^- = (0,471-/3-^/3) etc., where all three 
spin-wave modes are degenerate, allowing for a conve- 
nient comparison with the strong-coupling result. In the 
strong-coupling limit (U/t — > oo), the spin- wave energies 
are given by: 



c q = 3J5[(l-7 q )(l + 2 7q )] 



1/2 



(21) 



While the spin- wave energy wk at the MBZ vertices K 
vanishes along with the gap at U ~ 5, the spin- wave en- 
ergy ojm at the mid-points (M) of the MBZ edges shows 
evidence of competing interactions. Figure 3 shows the 
behaviour of lum with U for the soft, non-degenerate 
mode (see Fig. 1), along with the strong-coupling re- 
sult 2JS. With decreasing U, ojm falls rapidly and van- 
ishes at U* M = 6.8. For U < U^, the maximum eigen- 
value of [x (qM)] exceeds 1/U, signalling an instability 
of the 120° ordered state. This instability is driven by 
out-of-plane fluctuations, as the corresponding instabil- 
ity eigenvector <^ has non- vanishing amplitudes only for 
spin direction fj, = z, with identical magnitudes for all 
three sublattices a — A,B, C. Analysis of the fluctu- 
ation amplitudes in the instability eigenvector indicates 
spin twisting in the +z and —z directions along alternat- 
ing spin chains in the three lattice directions, implying 
instability towards a F-AF state, as in the square-lattice 
AF with NNN hopping t' . 

The spin-wave dispersion u; q along the K-M-K direc- 
tion [Fig. 4] shows a cross-over from a quadratic be- 
haviour around M for U > U^j to a linear behaviour as 
U -»• U 



A I 



6.8. For U < Um, there are no spin- wave 
where 7 q = ^[cosq x + 2cos(fe/2) cos(q y V3/2)], and the solutions near M as A q > 1/U. Despite the instability at 
three modes correspond to momenta q, q + Q, and U = U%. f , the fluctuation remains finite as U — > U^. This 
q — Q. As 7 q = for all three modes for <\k = 
(0,4tt/3V3), the spin-wave modes are three-fold degen- 
erate, with uj k = 3JS = 6t 2 /U for S = 1/2. Figure 2 
shows the variation of u>k with U, along with the strong- 
coupling result for comparison. With decreasing U, the 
spin- wave energy turns over, and the decreasing band 
gap effectively squeezes the spin-wave spectrum. Indeed, 
both lok and the band gap vanish together at U ~ 5 
(A = 2). This is a typical weak-coupling dynamical ef- 
fect for an itinerant antiferromagnet^ The divergence in 
the eigenvalue A q (o;) of the [x°(q, w)] matrix in Eq. (17), 
when spin-wave energy approaches the band gap, not 
only limits the spin-wave spectrum to within the band 
gap, but also strongly suppresses the spin-wave ampli- 
tude due to wave-function renormalization. 



is because the magnon dispersion cj q near M becomes 
linear in (small) momentum difference q = q — q^/ , and 
the fluctuation contribution of these modes (J qdq/cq) 
remains finite, just as for the Goldstone modes. A dis- 
continuous magnetic transition at U = is therefore 
clearly indicated. 

Turning now to long-wavelength modes, we consider 
the two spin-wave velocities (c = ojq/q) corresponding 
to in-plane (cm) and out-of-plane (cj_) fluctuations. For 
q = 0, the off-diagonal matrix elements of [x°(q, oj)], in- 
volving in-plane (/i = x, y) and out-of-plane (v = z) spin 
indices, vanish identically, implying no mixing between 
the in-plane mode of the x — y sector and the two out- 
of-plane modes of the z sector. Mixing is negligible for 
q << 1 as well, and therefore small-g spin- wave modes 
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FIG. 4: The spin-wave dispersion oj q along the K-M-K di- 
rection, showing the cross-over from quadratic to linear be- 
haviour around M as U — + — 6.8. 
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U 

FIG. 5: The spin- wave velocities for in-plane (||) and out- 
of-plane (_L) fluctuation modes, showing that c± vanishes at 
U ~ 6. The ratio approaches y/2 in the strong coupling limit. 

can also be identified in terms of in-plane and two out- 
of-plane fluctuations. In view of the exact degeneracy 
along the T — K direction, it is convenient to consider 
q = (0,<7j,), as the two spin- wave energies then readily 
yield the two spin-wave velocities, as described below in 
the strong-coupling limit. 

For q x — 0, the two modes q± Q yield identical values: 
7 q = — ^[l/2 + cos(<7 y \/3/2)], and hence degenerate spin- 
wave energies from Eq. (21). For small q y , one obtains 
w q = 3JS(^/3/2)q y and 3 JS(VS/2V2)q y for the modes 
q and q ± Q, respectively, yielding spin- wave velocities 
C || = 3J5(V3/2) and c± = 3 JS"(v / 3/2 v / 2), which are in 
the ratio ci|/c_i_ = 

Figure 5 shows the two spin-wave velocities in the full 
U range. Both velocities decrease as 1/U for large U, 
and the ratio C||/cj_ asymptotically approaches a/2 in the 
strong-coupling limit. The intermediate- U behaviour for 
the spin-wave velocity corresponding to out-of-plane fluc- 
tuations is most interesting, which exhibits a broad peak 
and falls rapidly with decreasing U, vanishing at a critical 
interaction strength U* tiS ~ 6, which is in the insulating 



FIG. 6: Enhancement in c± with hole doping, due to a sup- 
pression of the intrinsic frustration in the triangular lattice. 

regime. For U < U* m , the 120° AF state is therefore 
unstable with respect to out-of-plane fluctuations. 

The vanishing spin-wave velocity and spin stiffness 
(p± oc CjJ implies that the first-order quantum correc- 
tion to sublattice magnetization due to transverse spin 
fluctuations diverges. Therefore, the corrected sublat- 
tice magnetization will vanish at a somewhat higher crit- 
ical interaction strength U* ldcl . > L^* tiff , where the quan- 
tum reduction due to fluctuations exactly eliminates the 
mean-field order. Hence there is a quantum phase tran- 
sition in the triangular-lattice AF at U = U* rdcr from 
a 120° ordered state to a spin-disordered state. As de- 
scribed below, this QPT is driven by finite [/-induced 
competing interactions and frustration. 

In the strong-coupling limit, the Hubbard model at half 
filling maps to the 5 = 1/2 QHAF with NN interactions. 
However, for finite U, extended-range spin couplings (of 
order t^/U 3 and higher) are generated. Within the RPA 
analysis, the spin couplings are approximately given by 
Jjj re J7 2 [x°(w = 0)]ij, and have been obtained for the 
square lattice from a systematic t/U expansion^ Now, 
for the unfrustrated AF, the 0(t 4 /U 3 ) couplings be- 
tween sites separated by two hoppings (NNN and NNNN) 
are ferromagnetic, which actually stabilize the AF state 
against the i'-induced frustration.^ However, for the 120° 
state on the triangular lattice, the extended-range ferro- 
magnetic couplings are a source of additional frustration, 
leading to spin softening. 



V. HOLE AND ELECTRON DOPING 

The two AF bands in the 120° state are quite asym- 
metrical, with very different Fermi surfaces for hole and 
electron doping, suggesting quite different behaviour. In- 
deed, we find that while hole doping stabilizes the AF 
state and actually increases the spin stiffness, any elec- 
tron doping destroys AF ordering. 

Stability of the square-lattice AF state for hole and 
electron doping was studied earlier within the t — t'- 
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tive, indicating instability of the 120° AF state for any 
electron doping. Furthermore, the spin stiffness is seen to 
be quite independent of doping concentration. Very sim- 
ilar results for the t— t'-Hubbard model were obtained for 
electron doping in the elliptical electron pockets around 
k = (±7r/2,±7r/2)^LS The large relative magnitude of 
the intraband contribution is characteristic of the highly 
elliptical Fermi surface which reduces the energy denom- 
inator in the intraband particle-hole process. 



VI. CONCLUSIONS 



FIG. 7: Spin-stiffness contributions corresponding to total 
(interband+intraband) and interband particle-hole processes, 
for A = 3. The spin stiffness increases with hole doping (solid) 
whereas it becomes negative for electron doping (dotted). 



Hubbard models For finite doping, the Fermi energy 
lies within a band, and a key role is played by the in- 
traband particle-hole processes in Eq. (17) for [x°(q, w)], 
which generate additional frustrating spin couplings and 
affect the stability of the AF state with respect to trans- 
verse spin fluctuations. For positive t' , the AF state is 
destroyed for any electron doping, while a finite concen- 
tration is required for hole doping. 

We find a similar doping behaviour for the triangular 
lattice. For hole doping in the (broadened) lower AF 
band in a circular hole pocket around k = (0, 0), the spin- 
wave velocity c_l initially increases (Fig. 6), and then 
rapidly falls to zero with further hole doping, indicating 
destruction of the AF state. The initial enhancement in 
cj_ is due to a suppression of the intrinsic frustration in 
the triangular lattice, as explained below. In order to 
highlight the role of doping, we have kept A rather than 
U fixed, so that the band aspects remain unchanged with 
doping. Over a small doping range U remains nearly 
same, as shown in the figure. 

Figure 7 shows the spin-stiffness contributions corre- 
sponding to total (interband + intraband) and interband 
particle- hole processes in [x°(q, ui — 0)]. Here A = 3 and 
the U values range between 7 and 8. Interestingly, hole 
doping is seen to enhance the interband contribution, re- 
flecting an effective suppression of the intrinsic frustra- 
tion by eliminating the contribution of long-wavelength 
states in the lower AF band. While the intraband con- 
tribution (total — interband) does tend to destabilize 
the AF state (negative stiffness), the dominant interband 
contribution causes a net stabilization. A similar linear 
increase in the interband contribution with hole doping 
was obtained in a detailed analytical study of the t — t'- 
Hubbard models 

In contrast, for electron doping in the (narrow) upper 
AF band in elliptical electron pockets located symmetri- 
cally on the MBZ edges at k = Q/2 = (±tt/3, ±7r/\/3) 
and (±27r/3, 0), the spin stiffness abruptly turns nega- 



In conclusion, spin-wave excitations were studied in 
the 120° AF state of the Hubbard model on a triangu- 
lar lattice at half filling as well as for electron and hole 
doping. The triangular-lattice antiferromagnet presents 
a novel case of {/-controlled competing interactions and 
frustration, in contrast to the square-lattice case where 
frustration arises from the NNN coupling generated by 
the hopping term t' . The spin- wave energy com van- 
ishes at Utj = 6.8, and the instability of the 120° or- 
dered state against out-of-plane fluctuations implies a 
first-order quantum phase transition in the insulating 
state which pre-empts the vanishing spin-stiffness in- 
stability involving divergent quantum spin fluctuation 
and a continuous phase transition. The suppression of 
magnetic ordering due to enhanced quantum spin fluc- 
tuations arising from the finite [/-induced frustration 
provides an explanation for why no long-range mag- 
netic ordering is seen in the nearly isotropic triangular- 
lattice antiferromagnet k - (BEDT - TTF) 2 Cu 2 (CN) 3 . 
Furthermore, the realization of a non-magnetic insulat- 
ing state at intermediate U, which allows for a non- 
magnetic insulator - paramagnetic metal transition when 
the AF band gap vanishes with decreasing U and the two 
bands start overlapping, is relevant for the layered sys- 
tem k - (BEDT - TTF) 2 Cu[N(CN) 2 ]Cl, which exhibits 
a Mott-type metal-insulator transition not accompanied 
by any magnetic symmetry breaking. 

A highly asymmetric doping behaviour was obtained 
for electron and hole doping, when the intraband particle- 
hole processes were incorporated in the spin-fluctuation 
propagator. While hole doping was found to initially sta- 
bilize the AF state and actually increase the spin stiff- 
ness by suppressing the intrinsic frustration, any electron 
doping was found to destroy the 120° AF ordering. In 
fact, the stabilization for hole doping and instability for 
electron doping, with nearly concentration independent 
negative spin stiffness, closely resembles the doping be- 
haviour of the square-lattice AF within the t— t'-Hubbard 
model for positive t! . 

Finally, whether a non-magnetic insulator intervenes 
between the paramagnetic metal and the AF insulator 
depends on the relative strengths of the quantum cor- 
rections to magnetic order and AF band gap. In case 
the AF band gap closes before the magnetic order is lost, 
the magnetic transition is pre-empted, leaving a (nearly) 
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first-order AFI - PM transition, when the AF bands start Acknowledgement 
overlapping, possibly with an intervening AFM phase in 
a narrow U range. In this context, quantum corrections 
to quasiparticle dispersion and band gap due to motion 
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